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The collection and dissemination of educational statistics have been a 



changed considerably since then and the demands made for the scope, depth and 



federal responsibility for almost 120 years. The federal role in education has 
accuracy of information have increased accordingly. While the appropriateness 



of the federal government's role in the collection and provision of statistical 
informatio!) is not seriously questioned, debate about what and how data should 
be collected is recurrent. The reason for this is that statistics are not ends 
in themselves — they support and facilitate the attainment of other purposes. 
Many of these purposes are controversial and involve real or potential conflict 
not only within the federal government but among different levels of government. 
There seem to be four broad purposes for which educational statistics are used: 



1) To provide information on conditions and trends relating to the 
characteristics and performance of learners and educational 
institutions, 

2) To guide the allocation of resources, 

3) To assist in the enforcement of laws and regulations assuring 
students freedom from racial, ethnic, gender or other invidious 
discrimination, and 

4) To facilitate the improvement of educational policies and practices. 
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Until recently, most uses of ^ educational statistics related to the first 
three of these purposes. However, the current surge of interest in educational 
reform has seen policymakers and reformers turn to educational statistics to 
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determine how effective schools have been and what Improvements might result In 
the greatest Increases In student learning. And, as mo^e and more people have 
sought to put educational statistics to work In the cause of change, the limits 
of available Information have become more apparent. As Cooke, Glnsburg and 
Smith (1985) point out, Information available from the National Center for 
Educational Statistics (NCES) Is often Inaccurate, Incomplete and Inconsistent. 
NCES has no monopoly on such credits. In particular, the extant Information 
tells us too little about the outcomes of education. And, when such Information 
Is available, measures that might account for differences among students and 
school systems often are not. 

This paper focuses on how educational statistics might better serve the 
quest for educational Improvement In elementary and secondary schools. In 
seeking answers to this question, I briefly Identify the contributors to the 
federal Information base and suggest that available resources could be better 
used. I then turn to the notion thac If we want statistics to serve the 
Interests of educational Improvement, we need to conceptualize the sources and 
processes of school productivity. Such an analytical model Is presented and the 
types of questions suggested by this framework for which statistical Information 
could provide answers are Identified. 

Some Implications of this approach for current studies undertaken by 
federal agencies, especially the National Center for Educational Statistics 
'NC3S), and for future Inquiries, are explored. I conclude by arguing that the 
quality of educational statistics can be Improved If (a) statistical surveys are 
theory driven and policy focused and (b) the use of statistics Is made easier 
and more productive. 
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THE RELATIONSHIP BETWEEN EDUCATIONAL STATISTICS AND EDUCATIONAL RESEARCH 
The collection of educational statistics should both be Informed by and 
facilitate research. Statistics, by themselves, seldom tell a very rich story 
but they can and should encourage better story telling. 

I use the term research here to mean the process by which new knowledge 
that explains why things happen Is discovered. To believe that one can 
understand what Is going on In schools by knowing the characteristics of 
schools, even If one knows the performance of students In those schools. Is to 
be presumptuous or naive — or both. 

Statistical gathering and analysis that Is uninformed by research 
encourages simplistic conclusions about the causes of student performance. 
There are two reasons for this. The first Is that Important variables may be 
omitted from the data sets while Irrelevant data may be collected. Requests for 
Information that few people will or could use makes unproductive r^-^mands on data 
providers but the omission of variables encourages Inappropriate analysis. A 
second characteristic of statistical Information the collection of which Is not 
Informed by research Is that key variables may be Inappropriately specified and 
measured. 

Data that have been shaped by research and collected with theoretical 
concerns In mind obviously facilitate research. Both the extensive utilization 
of the National Longitudinal Study of the High School Class of 1972 (NLS-72) and 
the limitations of these analyses makes this point. In comparison to the NLS* 
72, the High School and Beyond (HSB) Study was much better Informed by ref iarch 
and theory and the potential payoffs from analyses of these data appear to be 
enormous* 

•i 
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Financial resources available for educational research are meager. At the 
same time, there Is a virtual army of actual and potential educational 
researchers that could attack "'research-Influenced" Information on education In 
search of new knowledge. There are ways that the energies of this army could be 
enlisted and redirected from the ubiquitous mall surveys and meaningless 
descriptive studies but an exploration of such strategies seems to be the topic 
of another paper. 



Research based on the types of statistical Information typically collected 
by government agencies or their contractees, even data as rich as those 
collected In the HSB Study, can tell only part of the story. Analysis of large 
scale stfitlstlcal data should lead to and be Informed by more Intensive research 
that examines the dynamics of student and teacher Interaction and otherwise 
helps us to avoid false assumptions about the meaning of statistical data. For 
example, the Introduction of comprehensive teacher evaluation plans, especially 
those that Involve standardized statewide criteria and processes. Is one of the 
most far*reachlng reforms ever Introduced In American schools. But It Is not 
the presence of these plans or even the putative characteristics of the plans 
that will explain any changes In teacher behavior; It Is the way these plans are 
Implemented and the data from them are utilized that will tell the tale. One 
cannot determine how an evaluation system (a curriculum, or other Innovations 
such as a new Instructional method, or a management practice) Is Implemented and 
how context affects Implementation without using research techniques that are 
more Intensive than statistical surveys. 



In summary, research and statistical surveys should be seen as having a 
symbiotic relationship. This Is particularly true with respect to statistics 
that might Inform decisions about school Improvement. 



Er|c 



THE UNDERUTILIZATION OF EXISTING CAPACITY 



The National Center for Educational Statistics collects, directly and 
Indirectly, only a portion of the Information that might be relevant to 
educational Improvement. Within the Department of Education, a host of other 
data collection efforts are regularly undertaken. These Include various policy 
studies and program evaluations conducted or supported by the Office of Planning 
Budget and Evaluation; research and Information collected by line agencies such 
as the Office of Special Education and the Office of Bf lingual Education and 
Minority Language Affairs; the National Center for Research on Vocational 
Education and thp. National Institute for Research on the Handicapped; The Office 
for Civil Rights; and various programs within the National Institute of 
Education, including the National Assessment of Educational Progress (NAEP). 
This list is by no means exhaustive. The point is that an enormous amount of 
informati n is collected on American education but there is no central effort to 
plan or coordinate the information collected or even to consolidate it once it 
is collected. This is not to argue that all data collection efforts within the 
Department of Education should march to the same drummer. But, the virtual 
absence of past efforts to define the information needs of the educational 
improvement mission of the Department of Education denies the Department 
opportunities to inform the country and Influence policy. Various mechanisms 
have been employed over the years to -control" data gathering efforts but these 
have been focused primarily on the reduction of paperwork and other burdens 
imposed on data providers rather than on improving the quality and accessibility 
of information. 



Given the absence of efforts to plan and coordinate data collection and to 
consolidate statistical data within the department, it is not surprising that 
the education-related data collection efforts undertaken by other federal 
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agencies are Dot part of a comprehensive plan and that data are not assembled In 
one place. Among the agencies that regularly collect Information that might be 
helpful In understanding and facilitating school Improvement are the National 
Science Foundation, the Department of Labor, the Department of Health and Human 
Services, the Department of Commerce (especially the Bureau of the Census), and 
the Department of Defense. 



The Secretary of Education could take the Initiative In designing a master 
plan for education-related statistics that would encompass the statistical 
activities of all of the agencies Identified above. The first step In that 
regard would be to catalog current and planned programs. The second step would 
be to Identify the key variables upon which major studies focus and the uses to 
which the data are part. A logical outcome of such an Initiative would be the 
establishment of a Federal Interagency Advlsoiy Committee on Educational 
Statistics. Such a committee could be staffed from the Office of Educational 
Research and Improvement. An Interagency effort could (a) Identify sources of 
data (b) suggest how existing data can be Integrated (c) Identify areas of 
unnecessary hindrance and Important Issues about which data are needed and (d) 
provide advice to the developers of major new efforts to collect educational 
Information. * One difficulty that will confront such an effort Is that the 
Department of Education Is a minor player In the statistics collection game; It 
does not have a lot to trade with when It seeks cooperation. But the interest 
In education Is great now and, at least for the short term future, the 
possibility for collaboration may exist. 



^ NCES has regularly Included representatives from the Census Bureau and the 
Bureau of Labor Statistics on its advisory board and this seems to have been 
constructive. 
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States and localities generate much of the data collected by the federal 
government. In addition, states and "localities collect a great deal of 
Information that does not now become part of the federal data base. The growth 
In the capacity of noo-federal education agencies to generate and analyze data 
appears to have been extensive In the last few years. These ''unassembled'* data 
have potentially significant uses In fostering school Improvement but this 
potential Is undermined by the absence of standardised definitions of key 
variables, variation In the Information collected from state to state, 
differences In data collection processes and the difficulty of retrieving the 
data collected. 

A good example of the weaknesses of current state level data Is provided by 
Information about teacher supply and demand. At a time when teacher shortages 
are widely anticipated, states presumably would find It valuable \^ know how 
their policies shaping supply and demand compare In effectiveness to those of 
other states. Some comparative data on state policies are available (cf. Roth 
and Mastaln, 1984). However, data on the consequences of these pollcles~e.g. 
numbar of certified teachers seeking employment, the qualities of these teacher 
candidates, who actually enters the profession, the volume of reentry, and the 
rate and character of teacher attrition — are hard to come by even though Such 
data are available for states to secure. 

The federal government could take the lead In encouraging and even 
assisting In the collection of policy-relevant Information on the supply and 
demand for teachers. One mechanism for planning such an Initiative Is the not 
overworked Intergovernmental Advisory Council on Education. NCES could be 
responsible for Implementation. 
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A THEORETICAL FRAMEWORK FOR SPECIFYING STATISTICAL DATA 
RELATED TO THE IMPROVEMENT OF SCHOOLS 
Like data analysis, data collection related to school improvement should be 
guided by two central questions: 

1. What are the problems or Issues we want to understand better? 

2* What theory or theories might best Identify the range of factors that 



If the answer to the first of these general questions Is to know how to most 
cost-effectively Improve schools, the answer to the second question depends on how 
we would decide a school had Improved. I suggest that the central purpose of 
schools Is to produce student learning. Thus, Improvement would be measured In 
terms of amounts or rates of learning or, at least. In terms of the relative 
achievement of students once all of the variables schools cannot Influence are 
taken Into account. I will return below to the types of learning In which we might 
be Interested. 

OVERVIEW OF THE MODEL 

The effectiveness of most organizations that are In the business of producing 
something Is the consequence of the Interaction of five types of factors: 

o the clarity and specificity of goals being pursued (e.g. student academic 



Influence the outcomes In which we are Interested? 



achievement). 



o 



the nature of the raw maf'erlals (e.g. the potential for learning students 



bring to school). 



o 



the sophistication and appropriateness of the technology or means of 



production (e.g. the curriculum), and 



o 



the quality of the craftmanshlp (e.g. teacher behavior) 



o 



""environmental** conditions that facilitate or Impede 



productivity 
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within this framevorki the specific variables to be measured will have to be 
Identified. I will suggest a number of the key factors that current research 
suggests accounts for variation In student achievement from classroom to classroom 
and from school to school. This model Is based on an extensive recent review of 
research syntheses and Individual studies (Uawley and Rosenholtz, 1984). The 
relative Importance of these factors for enhancing student achievement has been 
affirmed by a report urging school reform prepared by thlrty*two school 
superintendents from around the country (National Consortium on Educational 
Excellence, 1984). 



There are, of course, other efforts to synthesize current research on 
school effectiveness (cf. MacKenzle, 1983; Brophy and Good, forthcoming; Purkey 
and Smith, 1983; Northwest Regional Education Laboratory, n.d.)* While there Is 
substantial overlap In the conclusions reached In these reviews, there are also 
differences. Moreover, the model presented below Is derived from research on 
student achievement measured In conventional terms (l«e«, tests of skills and 
knowledge) and that Is a significant limitation. The ""Learning Productivity 
Model** outlined here should be considered Illustrative. It seems to be, 
however, a useful starting point In Identifying the types of statistics that 
will be most helpful In understanding and facilitating school Improvement* 



Because scholars disagree about the major determinants of school 

effectiveness and because the Importance of different factors depends on the 

outcome of schooling with which one Is concerned, the model or models which 

would guide federal data collection and coordination efforts should be derived 

from a consensus of leading researchers and practitioners. The process by which 

such consensus might be developed Is not difficult to Imagine: experts should 

be selected by a snowball reputatlonal technique and they should be provided 

with alternative assumptions about variables and their Interrelationships. The 
o .J 
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process should be Interactive until agreement Is reached either about particular 
variables or alternative explanations for specified outcomes- Thus, I have 
chosen not to try to detail specific variables about which one might want 
statistical data but instead to pose questions which direct attention at the 
types of data that would be useful In assessing and fostering school Improvement. 

The Learning Productivity Model of school effectiveness focuses attention 
on schools and classrooms. This seems an appropriate strategy for organizing 
one's thinking about school Improvement given the substantial evidence that 
change that affects the quality of education children experience must be 
generated at the school level If It Is to have a significant and continuing 
Influence (Hawley and Rosenholtz, 1984, Ch. 1). Moreover, the frequency with 
which one sees schools of widely different quality within the same district, 
even If one takes Into account differences In student body characteristics, 
reinforces the Idea that school Improvement-related data should provide well 
textured pictures of schools. However, a number of environmental conditions 
create the context for school Improvement and need to be considered. 

The types of factors that Influence school productivity and their dominant 
interrelationships are summarized In Figure 1. Table 1 Identifies key variables 
that comprise each of these factors. Before turning to a discussion of these 
variables and the directions they point to with respect to the collection and 
provision of statistical Information, the Issue of educational outcomes needs to be 
addressed. 

The national Interest In school Improvement obviously requires that we focus 
on the ability of schools to produce certain outcomes In the form of student 
learning. I want to distinguish between these school outcomes from the 
consequences of formal learning. I will refer to the latter, for lack of a better 

term as "lifetime outcomes". j j[ 
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SCHOOL OUTCOMES, There appears to be widespread agreement on the Importance 
we should attach to the acquisition of so-called -basic skills" &nd knowledge about 
the more or less traditional subject mattar measured by tests like the National 
Assessment of Educational Progress, the Achievement Tests and Advanced Placement 
Tests administered by the Educational Testing Service, and various standardized 
tests which report student performance in terms of grade x<.vel equivalencies. 

There Is growing concern, however, reinforced by the poor showing cf students 
on tests of so-called "higher order skills", that our schools are falling to teach 
students to reason Inf erentlally, to engage In reflection about and systematic 
analysis of complex problems, and to be creative In the face of uncertaln«"y. 
Assumptions about the Importance of these and similar capabilities to the life 
chances of Individuals and to the economic and social health of the society have 
yet to be tested. But, there can be little doubt that Interest In these types of 
school outcomes Is growing. There can also be little doubt that there is an 
absence of consensus not only about how to measure these outcomes but how to talk 
about them with shared meaning. It follows that little Is known about how 
variations In curricula and Instructional practices relate to these outcomes. 
Thus, the federal government could contribute to enlarging the definition of school 
effectiveness If It could facilitate the development of clearer conceptions of the 
outcomes by which the effectiveness of schools might be measured. The importance 
of such an effort, which could take the form of conferences and relatively simple 
studies of expert and leaders' opinions, is significant because it is possible— as 
Cooke, Ginsburg and Smith (1985) observe— that the factors that maximize student 
performance on tests of basic skills may not produce other types of student 
learning, such as "higher order skills". Indeed, some researchers believe that 
some elements of the technology of schooling that produces learning of the so- 
called basics may constrain other learning. ^ 
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LIFETIME OUTCOMES. Educators, policy-makers and parents have debated the 
priorities that schools bhould emphasize since there have been public schools. 
The nation seems to swing between different priorities with the various 
advocates Invariably making assertions about the effects that one focus or 
another will have on the experiences and well-being of students once they leave 
school (cf. Ravltch, 1983). Almost all of this advocacy, however, Is 
unburdened by evidence about the relationship between variations In school 
experiences and differences In students* post-school outcomes. 

While It might be argued that curricula should reflect the values of 
society, or the community, ^r the parents Independent of the extrinsic 
consequences they mlgtit have for students, this Is an unsatisfying position 
seldom asserted by the advocate of different currlcular emphases. Thus, it 
seems appropriate that the Information we have about America's educational 
systems Include data that would nllow us to understand how differences in 
content and levels of Investment lead to differences in outcomes beyond academic 
performance and educational attalment«> 

Two obvious educational outcomes that it seems useful to know about are 
occupational success and Income. The Importance of such Information is 
underlined by a recent synthesis of studies that concludes that individuals' 
academic performance (as measured by grade point averages or standardized test 
scorra) have almost no relationship to occupational success (Samson, et al., 
1984). The central theme of this review of research is reinforced by another 
recent study showing that a large national sample of employment officers place 
like emphasis when making hiring decisions place little emphasis ou most of the 
academic outcomes of education that schools are now being asked to focus upon 
more Intently (Grain, 1984). 

Other nonacademic outcomes of schooling that might help resolve some of the 
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recurrent debates about curricula and thus focus reforms on courses of study 
which benefit students are: 

o participation In the political and social 
life of communities 

o Incidence of antisocial behavior 

o family stability 

o condition of physical and mental health 
The Importance of considering the goals of schools and the priorities 
attached to them Is critical In understanding the potential for school 
Improvement In any given school or school system. The current educational 
reform movements, for example, rests on the assumption that the quality of 
schools has declined. The evidence supporting this assumption Is conflicting, 
however, (cf. Smith, 1984, Uawley, forthcoming) If we focused attention on how 
veil the schools did between 1970 and 1980 In (a) promoting basic skills, (b) 
meeting the special ne 3 of the disadvantaged and the handicapped, (c) 
Increasing the number of students who did well on Advanced Placement Tests, and 
(d) reducing the dropout rate and Increasing the number of students who attend 
at least some college, the schools would have to be judged moderately effective. 
If we ask how w( xl schools did In teaching Inferential reasoning and science, or 
developing students' understanding of democratic process, most schools would 
have to be judged to have declined In effectiveness. If the explanation for 
Improvements and declines have to do with how schools allocated time and other 
resources rather than how well they used them, the Implications for school 
Improvement would obviously be very different. This rather simple notion that 
the current dissatisfaction with schools Is the result of Increased and/or 
different expectations rather than declines In school quality has received 
almost no attention In the contemporary public discourse about education In 
part, perhaps, because there has been little evidence on how school outcomes are 
related to differences In the priorities given to particular goals and the way 
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these goals have been pursued* 



Recognlzlog that the factors that affect school productivity may vary 
depending on the outcome being focused upon, let me identify the general types of 
influences that current research suggests determine student learning. The 
statistical information that seems to be both useful and feasible to obtain about 
each of these influences is indicated by a set of questions that need better 
answers than we now have. Recall that the Learning Productivity Model has five 
sets of interactive variables — goals, raw materials, technology, craftsmanship and 
environmental conditions. Each of these will be considered in turn. I will not 
cite the literature that links these variables to student learning^ unless 
othervrise noted. The reasons for believing that these factors and others related 
to the questions posed are important determinants of school outcomes are documented 
elsewhere (Hawley and Rosenholtz, 1984). 

THE GOALS OF SCHOOL IMPROVEMENT 

RELATIONSHIPS BETWEEN COALS AND OUTCOMES. Orj^anizat ional goals typically 
focus on products or processes. For our purposes, product goals — the types of 
learning we want to produce — should be our concern. It might well be interesting 
to know the fit between process goals and the processes actually being utilized but 
if we know the latter, which we would if we collected data on the elements of The 
Learning Productivity Model, we will have this type of statistical Information 
about the processes that are relevant to school improvement. 
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FIGURE 1 

THE LEARNING PRODUCTIVITY MODEL - KEY FACTORS AFFECTING SCHOOL EFFECTIVENESS 



OPPORTUNITY ENVIRONMENT 
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gUPPORT ENVIRONMENT 
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TABLE I 



KEY ELEMENTS OF THE LEARNING PRODUCTIVITY MODEL OF SCHOOLING 

GOALS 

1. CONTENT 

2. CLARITY 

3. FOCUS 

4. CONCENSUS 

CHARACTERISTICS AND QUALITY OF 
RAW MATERIALS/STUDENTS 

1. COGNITIVE CAPABILITIES 

2. ACQUIRED KNOWLEDGE 

3. MOTIVATION 

4. OUT-OF-SCHOOL RESOURCES FOR LEARNING 

5. OUT-OF-SCHOOL CONSTRAINTS ON LEARNING 

THE TECHNOLOGY FOR PRODUCING LEARNING 

1. INSTRUCTIONAL STRATEGIES 

2. ORGANIZATIONAL ARRANGEMENTS (EXAMPLES: CLASS SIZE, 

3. CURRICULUM 

4. TIME 

5. LEARNING RESOURCES (EXAMPLES: TEXTS, MATERIALS, 
TECHNOLOGY, ETC.) 

CRAFTSMANSHIP 

1. ABILITY AND COMPETENCE 

2. CONDITIONS THAT FACILITATE THE USE OF COMPETENCE 

3. MOTIVATION 

ENVIRONMENTAL CONDITIONS 

1. OPPORTUNITIES 

2. RESOURCES 
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One rather straightforward definition of an effective school Is that It Is 
a school which achieves Its goals. In this sense outcomes and goals can be the 
same. In order to understand the sources of school effectiveness, however. It 
will be useful to know the characteristics of the goals to which school 
personnel subscribe. 

CHARACTERISTICS OF GOALS. There are at least four characteristics of the 
goals to which Individuals In a school might profess commitments that appear to 
be theoretically related to productivity. The first of these Is the Intensity 
with which people subscribe to particular priorities. For example, two people 
may say that reading achievement Is their first priority but one may feel that 
reading supersedes all else by several orders of magnitude. For another, both 
reading achievement and math achievement may have high priority even though math 
Is ranked lower than reading. Saying that both Individuals give their first 
priority to reading Is misleading. Most ways of assessing priorities, however, 
use simple ranking procedures. (What Is needed here is something like the 
''temperature scales'* developed In the voting studies conducted by the Institute 
for Social Research at the University of Michigan.) 



A second dimension of goals that is relevant to student learning is the 
clarity with which their advocates can describe them. How precise, in other 
words, are the outcomes being sought? Do seemingly similar goals subscribed to 
by different teachers have the same meaning for each teacher when the specifics 
are elaborated? 

A third aspect of goals about which data might be gathered is focus. 
Theodore Sizer may overstate it when he says that the three most important 
things in explaining school effectiveness are focus, focus and focus. But the 



point is well t<*fcen (cf, Peters and Waterman, 1983, vis-a-vis the importance of 
focus in private organizations.) In other words, how many high priority goals 
do teachers and administrators in a school want to accomplish? 

A fourth characteristic of goals which appears to influence school 
effectiveness is how widely shared the goals are. 

THE RAW MATERIAL OF EDUCATION - STUDENTS 

Students are the raw material with which schools work. Students bring with 
them to school different capacities, knowledge and motivation for learning and 
the effectiveness of schools cannot be measured unless these things are taken 
into account. Moreover, schools can shape their probabilities of success by 
influencing the learning readiness of students. Ideally, we would want to know 
something about the intellectual capacity of students but this is difficult to 
assess validly under any circumstances and seems beyond the reach of statistical 
surveys. Other factors that schools might influence or that could be Influenced 
directly by public policies are students' previous knowledge, students' 
motivation, the resources and assistance students have available out of school, 
and home or neighborhood constraints on learning. Thus, we might better 
understand the effectiveness of schools and the promise of different improvement 
strategies if we had data that spoke to the following questions. 

What was the performance of the students in their previous school or class 
with respect to the outcomes in which we are interested? What is the 
socioeconomic status of the students? What types of preschool learning 
experiences, if any, did students have? What types of learning-relevant 
interactions do students have with their parents, older siblijgs and other 
adults with whom they spend considerable time? What proportion of the students 
are fruu uiu^le pareut fauillies auu uu^ mauy have a ^c^rent at hoiae when school 



Is out? What does the school and teachers do to Involve parents in the 
education of their own children? How much and what kind of homework do students 
do and under what conditious cao they study at home? What kind of supports for 
and constraints on learning are there In the students' immediate neighborhood? 
How murh time do students spend watching television and what do they watch? 

THE TECHNOLOGY OF EDUCATION 

When students go to school they experience a range of structures, processes 
and learning resources. These "means of production" — which obviously vary 
considerably among states, school districts, schools and classrooms — make up the 
technology of schools. Of course, the effectiveness of the technology in 
producing student learning is significantly Influenced by how well it is used 
but we can make policy — and do — about what I am calling technology more or less 
indepeodeDtly of the policies we make that affect the quality and performance of 
teachers and administrators* 



Data collected by NCES and other federal agencies are typically short on 
information about what actually happens in schools. The HSB data go a long way 
in addressing this limitation of available statistical evidence but, of course, 
they tell us only about high schools at two points in time. The types of 
information about school technology that research suggests would be helpful in 
assessing school productivity and developing improvement strategies include data 
on instructional strategies, organization arrangements, curriculum, the use of 
time, and learning resources. Some of the questions we would want this 
information to address are set out below. 



INSTRUCTIONAL STRATEGIES, 
teaching, programmed instruction, 
mastery learning, peer tutoring 
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What mix of strategies — such ao whole class 
individualized learning, cooperative learning, 
and interactive teaching—is eipployed and how 
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does this vary by subject matter and learning objectives'' How often are 
students evaluated and how are decisions made about mo ent through the 
curriculum and grade levels? How high and how clear are standards of student 
performance and what types of rewards do students receive for meeting standards? 

ORGANIZATIONAL ARRANGEMENTS. What Is the class size with and without teacher 
aides? How many students are there In the school and how are they assigned to 
Instructional units (e.g. ^^^^^s or "houses'" or blocks)? What grouping and 
tracking procedures are employed? What Is "skewness" of student performance In 
particular classrooms? 

CURRICULUM. What subjects are taught, at what level of difficulty and for 
how many hours each day or week. Does the school have a core curriculum that Is 
well articulated across grade levels? Are the tests used to measure student 
performance articulated wlch the curriculum? 

TIME. How many hours of actual Instruction are allocated each week free 
from Interruption? How many hours do teachers teach each week? How many days 
each year do students attend school? 

LEARNING FIESOURCES What Is the condition and nature of the instructional 
facilities? What support systems do teachers have— library, volunteers, audio- 
visual, business-school linkages, etc.— and how often are they available and 
used? How many computers are available and how are they used? 

THE QUALITY OF THE CRAFTSPERSON 

A decade or so ago, researchers and pollcymakers~but not parents— asked 
questions like "do teachers make a difference**? The recent outpouring of 
research on school effectiveness leaves little doubt that both teachers and 
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administrators make a big difference In the quality of education students 
receive, (cf, Hawley and Rosenholtz, 1984, Chs, 1-3), For example, not only do 
teachers Influence how students learn by the way they Implement a curriculum, 
teachers also shape what students are taught (Biophy, 1980; Green and Harker, 
1982). Recent research also suggests that teachers significantly Influence 
parental Involvement In their children's education (Epstein, 1984). 

Statistical data about educators Is very limited. It Is not possible, for 
example, to get a clear Idea of what the career paths of principals and 
administrators look like under different circumstances. Little Is known about 
the qualities of those who actually teach or what difference these qualities 
make In student performance (cf. Evertson, Hawley and Zlotnlk, 1985). Very 
little Is known about teacher attrition. And so on. (The current activity 
within NCES to develop a better picture of teachers Is ambitious and well 
conceived and It will be much welcomed by policymakers. A similar effort with 
respect to school administrators Is also needed.) 

There are three Interrelated aspects of craftsmanship which appear to 
Influence student learning: ability and competence, conditions that facilitate 
or Impede the use of competence, and motivation. Some questions to which It 
seems Important to have answers about these Interrelated aspects craftsmanship 
follow. 

ABILITY AND COMPETENCE. What are the levels of general academic ability 
and subject matter knowledge that teachers and principals possess? How much of 
what types of formal training have teachers and principals had and at what stage 
of their career did this training occur? How many and what kind of 
opportunities for Increasing their professional competence have tearhers and 
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principals had? What opportunities have teachers and principals had to learn 
Informally from their peers? 



MOTIVATION, What priorities do teachers say the' ^.^ve to the different 
reasons why they teach? What Is the volume and character of feedback about 
their performances that teachers and administrators receive? Is there a formal 
evaluation system? What are Its criteria and how Is It used? What Is the 
salary range that Is accessible and on what bases are salary Increases awarded? 
Are there financial Incentives available other than salary Increases? Do 
teachers and administrators have an opportunity to participate In key decision 
affecting professional practice? In what ways Is superior professional 
performance recognized and rewarded ^besides economic rewards)? 

CONDITIONS THAT FACILITATE EFFECTIVE PRACTICE. What level of support do 
teachers and administrators receive from their respective administrative 
superiors? How often do teachers have the opportunity to Interact 
professionally with their peers and do norms of colleglallty exist? What Is the 
level of order and discipline In the school? Is the teaching time of teachers 
protected from Interruptions and diversion? How widely shared are goals for 
student learning and at what levels of expectation are these goals pitched? 
What are the number of students In the school? 

It should be noted that NCES has focused Increasing attention on teachers In 
recent months and several surveys have sought to collect data about teachers and 
what they do. While this effort addresses several of the questions raised above, 
the bits and pieces apparently cannot be aggregated and too little attention Is 
focused on outcomes. The HSB data do provide student outcome data but little 
Information about teachers* careers and personal characteristics. The NCES labor 
market survey provides Information about teacher shortages and teacher Incentive 
plans but no Information about salaries. «) t 
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ENVIRONMEOTAL CONDITIONS 

The problem of controlling for non-school factors that affect the 
productivity of a scnool cannot be dealt with adequately except under 
experimental conditions. Therefore, the theory that guides data collection must 
encompass critical student characteristics beyond the reach of the school and 
aspects of the schools' ecology. Much has been written about the f.ictors 
external to individual schools that influence school improvement. 
Unfortunately, aside from a handful of case studies, little empertcal research 
on the relative importance of these factors exists. Thus, unlike other 
dimensions of the Learning Productivity Model, this one is not researched-based. 

The collection of data relating to school external environments is 
problematic not only because so little evidence is available upon which to 
select from among the theoretically interesting ecological factors that could 
influence student learning but because of the difficulty of acquiring the 
information. If we were to constrain the collection of data to those that could 
be provided by occasional national samples or by school systems from existing 
information bases, the types of inforcation relevant to school improvement that 
might be collected seems a bit easier to think about. 

It seems reasonable to think of environmental conditions that influence 
school productivity as being of two broad types: (a) conditions that grant or 
constrain the opportunities of educators and students and (b) resources 
available with which educators and students can pursue their goals. 

THE OPPORTUNITY ENVIRONMENT. The opportunity environment essentially 
shapes ambitions and expectations and grants— formally or informally— authority 
to act. Its' dimensions and their consequences might be understood if questions 



ERIC 222 



25 



such as the following could be answered* How much decision-making discretion do 
school administrators and teachers have to make and Implement policy? How much 
support do the goals of school-level personnel have among dls trie t-level 
administrators and other teachers? How stable are residential patterns and 
pupil assignments to school. » How available to students are low-cost options to 
pursue postsecondary education? What types of of employment opportunities In 
the community can students expect to have upon graduation? 

THE RESOURCE ENVIRONMENT. The resources available to pursue the ambitions 
and expectations held by Individuals and by the collective professional and 
student populations of school should affect student learning* Resources, of 
course, create opportunities but the ""opportunl ty environment** just noted 
relates to sources of motivation provided by the environment while the **resource 
environment** provides capabilities that facilitate action relevant to the 
Individual and organizational purposes that have been discussed as deriving from 
factors that characterize the school and Its student body* 

The nature of the resource environment might be understood If answers were 
available to several questions* What Is the level of expenditures per pupil 
taking Into account the number of students with needs for resource-Intensive 
programs? What Is the quality and quantity of technical assistance available In 
the district to Implement school-level goals? How many Individuals participate 
In school activities as volunteers In support of Instructional or 
extracurricular activities? What resources, If any, are provided to schools by 
business and public and nonprofit agencies? What Is the **tax-ef f ort** (the 
extent to which the available tax base Is utilized) of the state and the school 
district? This last Indicator may be a surrogate measure of the school 
environment's commitment to public education* 
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SUMMARY 

I have argued that the collection of statistical data related to school 
Improvement should be guided by grounded theories and hvpotheses that might 
explain differences In student learning among schools and classrooms. A 
conceptual framework Implicitly embodying such explanations, which I have called 
the Learning Productivity MOvIel, has been outlined. Identifying the key 
elements of models and questions tnat might profitably guide data collection and 
analyses does not, of course, add up to specific recommendations about 
prl'^rltles that should be placed on gaining particular information that would 
facilitate the development of effective school imp )vement strategies. And, 
obviously, the ways such variables might be measured are only hii.ted at in the 
discussion above. My presumption, however, is that these next steps in the 
development of a plan for collecting statistical information is given direction 
by this model bulldipg exercise. 



CONCLUSION - NEXT STEPS 



Let me conclude by briefly discussing two courses of action which would 
result in statistical data bases that would support the formulation of 
productive school improvement policies and practices^ (a) the development of a 
plan for collecting new types of data (b) the integration and enrichment of 
exj^sting statistical information. 
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THE COLLECTION OF NEW DATA 

In general, statistical data related to educatir^n that is now available is 
not particularly helpful in developing new policy. The current array of 
educational statistics focuses primarily on the condition of education rather 
than on explanations for that condition. In other words, little effort has been 
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made to relate educational processes to educational outcomes. Moreover, as 
noted above, when outcomes are assessed, the range of educational outcomes 
measured Is narrow. For ^?xample, the 1985 Teacher Questionnaire, while It deals 
with how teachers spend their time, barely touchuj on the teacher 
characteristics, behavior, and circumstances most powerfully related to student 
learning and provides no outcome data whatsoever about either students or 
teachers. Of what value Is such Information? If It does lead somewhere, the 
probability that It will lead us down primrose paths Is at least as great as the 
prospect that the road It will direct us to will be paved with yellow brick. 



This generalization about the atheoretlcal character of data collection 
does not apply to all surveys* The most notable exception Is the HSB Study. 
There are lessons that this study has for new data collection efforts relating 
to school Improvement even though that Is not the primary purpose of the study. 
First, schools should be seen as social systems and both formal and Informal 
processes should be assessed. Second, an exj)ert review panel that guides the 
design and redesign of statistical studies seems to be cost-effective. Thus, the 
next step would be to convene such a panels refine the theoretical framework, 
specify variables, consider the sampling Issues and o*^ her wise develop the scope 
of the project so that tne potential benefits and the costs can be estimated. 
Ad HSB-type study will be very expensive but the return on Investment, In terms 
of usable knowledge and educational Improvement will probably be high. If early 
evidence on the use of the USB Study Is any Indication. Comprehensive data 
collection efforts like the HSB Study or a similar one which woul follow from 
the Learning Productivity Model would be much more valuable to policy makers and 
would-be reformers If data were collected that would allow analysts to 
understand the economic costs of alternative Improvement strategies* 




Because the statistical error that threatens all data analyses Is reduced 
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to the extent that one can take Into account all of the alternative explanations 
for a given phenomenon, and because we typically want large data sets to serve 
multiple purposes, an obvious problem faced by those who design statistical 
studies Is knowing what data NOT to collect* The potential cost/benefit ratio 
of data collection Itself Is one way to make such decisions and theory as well 
as long run utilization studies can be used to make such calculations. The 
probable reliability of the data Is another factor that should be considered and 
It would seem useful to develop, perhaps through expert Interviews, an 
understanding of the sources of error In survey responses that could be used 
across studies* For example, such considerations might Include the probability 
that Individuals would perceive themselves or a goal they value being advanced 
(or the converse) by accurate provision of data and the degree to which the 
Information Is readily available* 

THE INTEGRATION AND ENHANCEMENT OF EXISTING DATA 

I have argued that most statistical surveys provide only part of the 
picture we need to have of how schools operate and che outcomes they produce* 
This proposition holds for statistics that address Issues other than school 
improvement* There are good reasons for this having to do with cost, burdens on 
providers, etc*, but among the several studies conducted or sponsored by the 
federal government, there are many pieces of the puzzle* Moreover, several 
studies could be enriched if data available from published sources other than 
stati'^tical studies were added to various data sets* NCES, therefore, might 
consider the following strategies for making available statistics and 
recurrently conducted studies more accessible and more usable: 

(a) coordinate the data gathering of agencies within the 

Department of Education, across the federal government, 

and between federal agencies and other levels of government* 
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(b) "nest** future studies in such a way that data from the same 
sites could be Integrated* For example, It might be 
possible to conduct the HSB and National Assessment studies 
In the same or overlapping locations* 



(c) Identify conimon sites at which data has already been 

collected* For example, the teacher supply ,a demand study 
may have been conducted In locations for which data on 
teacher salaries were collected* 



(d) enrich existing statistical data by adding Information from 

other sources* For example, the teacher demand and shortage study 
could be enhanced by adding Information on career entry 
requirements (available from The National Association of State 
Directors of Teacher Educational Certification)* 



Steps such as these, coupled with efforts to prepare the data in formats 
that are well documented and readily usable by researchers and practitioners, 
would Increase the frequency and sophistication with which educational 
statistics were analyzed* The more analysis that Is carried out, the more Will 
be known about the strengths and weaknesses of the data collected* Such 
knowledge would result In Improvements In the quality of the data collected In 
future studies* 



SO, WHO WILL FUND ALL OF THIS? 

What I have called for Id this paper could easily be dismissed as being 
beyond the resources of the Department of Education and therefore unfeasible* 
Surely, anyone who suggests more federal activity these days would seem to be 
out of touch* There are two general ways to respond to the legitimate concern 
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about new expenditures. The first of these Is to do more by making better use 
of current resources. The second Is to convince policymakers that spending 
money on statistics will not only Improve learning but will facilitate more cost 
effective policy decisions. Let me consider the second of these arguments 
first. 

It Is worth noting that the Office of Management and Budget (0MB) often 
"discourages** statistics gathering proposals on the grounds that they l^^ck 
policy relevance. What Is being proposed here Is to Increase the relevance of 
statistics to policy making. 

All levels of government expend about $120 billion on public elementary and 
secondary education. The NCES budget Is an Infinitesimal proportion of this and 
If the costs of all other education data gathering efforts were added up, the 
relative level of Investment In statistics would still be minuscule. If the 
Information developed from the types of school Improvement-related data I have 
urged be collected were to better Inform the decisions of one large school 
system, the cost of the national effort might be covered. Among the types of 
decisions that might yield large dollar returns that could be Informed by the 
sophisticated analysis of quality data of the sort discussed above are choices 
about class size, beginning teacher salaries, the use of economic Incentives, 
Investments In traditional staff development and other matters. Of course, 
analyses could point us toward higher expenditures but. If so this would result 
In greater productivity. If Investments In schooling could be tied to post- 
school outcomes, the potential for return on the Investment In statistics would 
be very great. Of course, statistical data seldom provide, by themselves, clear 
policy directions. But they can challenge myths, they can suggest new options, 
they can confirm predispositions and they can Identify Issues worth further 
Investigation. Moreover, when cynics charge that statistics can be misleading 
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and often provide Incomplete pictures the answer might be: compared to what? 
Most policy choices are made on the basis of Intuitions Informed by 
predispositions and by the concern decisionmakers for their political and 
occupational futures. Good statistics can serve as a balance wheel, If not a 
steering wheel, In the policy process. 

What about making do with the resources we have? To pursue the course 
suggested In the heart of this paper. It would be desirable, perhaps necessary, 
to conduct one study a year of the size and sophistication of a **wave'* of the 
HSB Study. Multi-year longitudinal research does not seem necessary though, of 
course, that too would be valuable. Ooe possible source of those funds Is to 
discontinue work being done now that does not seem to go anywhere. In addition. 
It might be feasible to combine resources from several of the NIE centers whose 
missions overlap the purposes to which school Improvemen t-'relevant statistics 
coul d be put. 

It might also be possible, especially In view of the recent posl^.lon of the 
chief state school officers to use the NAEP data comparatively, to connect the 
study of school processes In some way to the outcomes being studied In the NAEP. 
This would probably require In a larger sample and Increased costs to conduct 
the NAEP but such plggy-backlng would reduce the overall expense of the school 
Improvement study even If this study bore the Increased costs of the NAEP. 

If a major new study or sec of studies was not possible, two other options 
remain. One Is for NCES to take the technical lead and provide some financial 
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Incentive that would encourage separate state studies that would be conducted, 
In part, In accord with a common design and with common basic variables being 
measured. A second option would be to do those several things noted earlier In 
the conclusion related to making more effective use of existing data. 

Where there* s a will, . . . 
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